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ABSTRACT 


A  new  measure  of  edge  information  for  color  images  based  on  cumulative  histo¬ 
grams  of  absolute  color  differences  is  proposed.  A  multispectral  version  of  the  Sym¬ 
metric  Nearest  Neighbor  filter  for  edge- preserving  smoothing  and  methods  for  image  seg¬ 
mentation  and  edge  detection  are  developed  based  on  this  measure.  Experimental 
results  show  that  the  performance  of  the  new  algorithms  is  very  good. 
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1.  Introduction 


The  analysis  of  color  Images  has  received  relatively  little  attention  In  com¬ 
puter  vision  research.  In  spite  of  the  facts  that  color  plays  an  Important  role  In 
human  vision  and  that  color  should  also  provide  much  useful  Information  for 
many  Image  analysis  applications^  Image  preprocessing  by  smoothing,  region- 
based  segmentation  and  edge  detection  are  among  the  basic  and  most  Important 
steps  In  most  applications.  A  wide  variety  of  methods  for  these  tasks  have  been 
developed  for  gray  scale  Images,  but  omy  a  few  for  color  Images. 

A  general  Introduction  to  color  Image  processing  and  analysis  Is  given  by 
Pratt  [l].  The  applications  of  human  visual  color  models  to  computer  vision  are 
reviewed  by  Granrath  [2]  and  Gershon  [3],  A  comparative  study  of  color  features 
was  carried  out  by  Ohta  et  al.[4].  Among  the  color  segmentation  approaches  pro¬ 
posed  In  the  literature  are  the  histogram  splitting  method  of  Ohlander  et  al.  [5] 
and  Its  successors  [6],  clustering  methods  by  Schachter  et  al.  [7]  and  Sarabl  and 
Aggarwal  [8],  and  a  method  based  on  edge-preserving  smoothing  by  Nagao  et  al. 
[9].  For  color  edge  detection  an  extension  of  the  Hueckel  edge  operator  was  pro¬ 
posed  by  Nevatla  [10]  and  a  mask  technique  by  Robinson  [ll].  A  color  smooth¬ 
ing  method  using  global  distribution  of  pixel  values  was  proposed  by  Kitchen  et 
al.  [12]. 

This  paper  presents  a  measure  of  edge  Information  for  color  Images,  which 
can  be  used  for  smoothing,  segmentation  and  edge  detection.  We  Investigate  the 
color  Information  In  histograms  of  first-order  color  differences.  A  multlspectral 


version  of  the  Symmetric  Nearest  Neighbor  filter  [13]  for  edge-preserving  smooth¬ 
ing,  and  methods  for  region-based  segmentation  and  edge  detection  are 
developed. 

These  methods  are  based  on  cumulative  histograms  of  absolute  color 
differences  computed  In  a  single  pass  through  the  Image.  Histograms  of  first- 
order  gray  scale  differences  have  been  used  by,  among  others,  Nagao  et  al.  for 
threshold  selection  [9]  and  Gotoh  et  al.  [14]  for  analyzing  properties  of  median 
filters  . 

In  this  paper  we  give  examples  of  our  approach  using  two-band  data.  The 
methods,  however,  generalize  straightforwardly  to  three-band  color  data. 

2.  Histograms  of  Color  Differences 

The  global  histogram  of  absolute  color  (or  gray  scale)  differences  provides  a 
good  measure  of  edge  Information  In  an  Image,  because  the  likelihood  that  an 
absolute  color  difference  occurs  In  the  Interior  of  a  region  decreases  monotonlcally 
with  Increasing  magnitude  of  the  difference.  The  general  form  of  the  histogram 
of  absolute  gray  level  differences  Is  depicted  In  Figure  1.  The  high  peak  on  the 
left  comes  from  differences  Inside  homogeneous  areas  and  the  small  peak  on  the 
right  from  edges  between  regions.  In  the  multidimensional  case  the  differences  at 
edges  are  sparsely  scattered  and  usually  no  significant  peaks  exist  In  the  histo¬ 
gram.  The  peak  caused  by  homogeneous  areas  Is  very  high  and  It  decreases  shar¬ 
ply  with  Increasing  absolute  gray  scale  or  color  difference. 


A  difference  histogram  Is  computed  In  one  pass  through  the  Image  by  consid¬ 
ering  absolute  differences  between  a  given  pixel  and  Its  four  neighbors. 

Instead  of  using  the  original  histogram,  we  can  easily  compute  a  cumulative 
histogram  of  absolute  color  differences,  In  which  we  can  be  certain  that  the  fre¬ 
quencies  are  monotonlcally  related  to  the  magnitudes  of  the  differences.  Here  the 
cumulative  frequency  associated  with  a  given  absolute  color  difference  Is  the  fre¬ 
quency  of  all  differences  which  are  greater  than  or  equal  to  It  In  each  coordinate. 
This  cumulative  frequency  corresponds  to  the  area  of  the  rectangle  In  the  original 
2-color  histogram  determined  by  the  given  value  at  Its  corner  (Figure  2). 

3.  Edge-Preserving  Smoothing 

The  Symmetric  Nearest  Neighbor  (SNN)  filter  recently  Introduced  by  Har¬ 
wood  et  al.  [13]  uses  both  spatial  and  nearest-neighbor  constraints  on  Image  pix¬ 
els  to  smooth  an  Image.  To  compute  the  gray  value  for  the  center  pixel  In  a  local 
neighborhood.  It  selects  half  the  number  of  pixels  In  .the  neighborhood  by  select¬ 
ing,  from  each  pair  of  pixels  located  symmetrically  on  opposite  sides  of  the  center 
pixel,  the  one  closer  In  gray  value  to  the  center  pixel.  In  case  of  tied  pairs,  the 
mean  of  the  pair  Is  used.  Then  the  mean  value  of  those  selected  Is  substituted 
for  the  original  value. 

To  find  symmetric  nearest  neighbors  for  a  multiband  Image,  the  following 


procedure  Is  proposed: 


1.  Compute  the  multidimensional  cumulative  histogram  of  absolute  color 
differences. 

2.  For  each  symmetric  pair  of  neighbors  In  a  local  neighborhood,  compute  the 
absolute  color  differences  between  the  two  pixels  In  the  pair  and  the  center 
pixel.  The  pixel  with  a  higher  frequency  In  the  cumulative  histogram 
(smaller  color  difference)  Is  selected.  In  case  of  ties,  the  mean  of  the  sym¬ 
metric  pair  Is  used.  The  mean  of  the  values  of  the  set  of  pixels  selected  Is 
assigned  to  the  center  pixel  on  each  band. 

The  color-SNN  Alter  can  be  Iterated  and  It  converges  without  producing 
artifacts;  normally  only  very  small  changes  occur  In  the  Image  after  2-3  Itera¬ 
tions.  The  hardware  Implementation  of  the  color-SNN  using  a  3  by  3  neighbor¬ 
hood  should  be  almost  as  straightforward  as  In  case  of  the  basic  SNN  Alter  [15]. 

The  histogram  of  differences  can  also  be  used  for  other  nearest-neighbor 
operations.  A  color  version  of  the  k-nearest  neighbor  Alter  [16]  can  be  Imple¬ 
mented  by  selecting  the  k  neighbors  with  the  smallest  color  difference  (highest 
values  In  the  cumulative  histogram)  for  averaging. 

4.  Segmentation 

Our  method  of  color  segmentation  combines  edge-preserving  smoothing  with 
a  simple  connected  components  (CC)  algorithm.  In  which  adjacent  pixels  are  said 
to  be  connected  If  the  likelihood  or  frequency  of  the  color  difference  Is  large  (so 
the  magnitude  of  the  difference  Is  small).  The  algorithms  make  use  of  the  com- 
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blned  Information  In  the  two-band  Image.  A  gray-scale  version  of  the  CC  algo¬ 
rithm  Is  described  In  [17];  both  versions  are  modifications  of  the  usual  one  for 
binary  Images  [18]. 

This  approach  Is  similar  to  that  of  Nagao  et  al.  [9],  except  that  they  used  a 
different  edge-preserving  filter  [19]  to  smooth  each  band  separately.  Then  small 
level  regions  of  the  separate  bands  were  merged  depending  on  multiple  thres¬ 
holds,  one  for  each  band,  to  determine  the  connectivity  of  4-adJacent  neighbors. 
This  method  performs  fairly  well,  although  It  has  some  weaknesses.  The  smooth¬ 
ing  filter  tends  to  distort  edges  and  to  create  artifacts;  also,  because  the  bands 
are  separately  smoothed,  there  are  often  inconsistencies  between  bands,  which  are 
especially  noticeable  at  region  boundaries.  In  addition,  there  Is  the  problem  of 
selecting  proper  thresholds  for  the  separate  bands. 

Now  we  will  describe  the  present  approach  to  color  segmentation  which 
minimizes  the  weaknesses  of  Nagao's  method. 

First  the  Image  Is  smoothed  by  the  eolor-SNN  filter.  Normally,  2-3  Itera¬ 
tions  of  3  by  3  filtering  are  needed  to  sharpen  edges  and  smooth  homogeneous 
areas.  To  make  edges  even  sharper  and  to  avoid  mlsmerglng  of  regions  at  some 
critical  points,  bands  are  edge-enhanced  with  a  gray-scale  filter  (MINRANGE) 
using  a  modification  of  the  “least  variation”  principle  of  Nagao  and  Matsuyama's 
smoothing  algorithm  [19].  The  MINRANGE  filter  replaces  the  center  value  of  a  3 
by  3  neighborhood  by  the  mean  of  the  4-plxel  "corner”  subgroup  (of  the  eight 
such.  Including  the  center)  having  the  smallest  range.  Because  sharpening  Is 


applied  to  almost  completely  smoothed  bands,  no  artifacts  are  generated.  More 
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discussion  on  sharpening  and  on  recent  studies  with  single-band  SNN  smoothing 
Is  presented  In  [20]. 

After  the  color  Image  Is  smoothed,  It  Is  segmented  by  a  two-pass  connected 
components  (CC)  algorithm,  In  which  adjacent  pixels  are  said  to  be  connected  If 
the  likelihood  or  frequency  of  their  absolute  color  differences  Is  sufficiently  large. 
The  only  parameter  Is  a  threshold,  expressed  as  a  centlle  of  frequencies,  which  Is 
supplied  by  the  user. 

First,  the  two-band  histogram  of  absolute  color  difference  Is  computed, 
which  Is  first  converted  to  a  two-band  histogram  of  cumulative  frequencies  and 
finally  to  centlles  of  their  distribution. 

The  two  passes  of  the  CC  algorithm  are  the  same  as  those  of  the  standard 
one.  Row  by  row,  pixels  are  assigned  labels  by  comparing  each  pixel  with  the 
four  adjacent  pixels  above  or  to  the  left,  which  have  already  been  labeled  as  the 
Image  Is  scanned  from  top  to  bottom,  left  to  right.  Then,  In  the  second  pass,  the 
pixels  with  component-equivalent  labels  are  relabeled  uniquely. 

5.  Edge  Detection 

In  this  section  we  describe  a  method  for  detecting  color  edges  using  the 
difference  histogram  approach. 

The  magnitude  of  a  color  edge  at  a  pixel  might  ordinarily  be  defined  as  the 
root- mean-square  of  the  normalized  band  differences  [21]. 
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Here,  after  possibly  smoothing  the  Image  with  the  color-SNN  and  MIN- 
RANGE  Alters,  the  cumulative  histogram  of  absolute  color  differences  Is  com¬ 
puted  and  converted  to  centlles  of  the  distribution  of  cumulative  frequencies. 
Then  the  color  edge-value  at  a  pixel  Is  the  maximum  centlle  of  the  cumulative 
frequencies  of  the  color  differences  between  the  center  pixel  and  Its  eight  neigh¬ 
bors.  By  thresholding,  we  obtain  a  binary  edge  Image. 

6.  Experiments  and  Discussion 

Figure  3  shows  the  red  and  green  bands  of  a  color  Image  of  a  house.  The 
size  of  the  Image  Is  255  by  192  pixels  with  eight  bits  per  pixel.  Figure  4  shows 
the  same  bands  after  two  Iterations  and  Figure  5  after  five  Iterations  of  color- 
SNN  smoothing.  The  Images  become  much  smoother  (note  the  tree)  and  the 
edges  are  well  preserved.  The  difference  between  Figures  4  and  5  seems  to  be 
small. 

A  good  way  to  evaluate  the  effectiveness  of  smoothing  Is  to  consider  Its 
effects  on  segmentation.  Because  our  CC  segmentation  Is  very  sensitive,  depend¬ 
ing  on  single  linkage  of  adjacent  pixels,  the  results  will  be  poor  If  the  Image  Is  not 
well  smoothed  while  enhancing  edges  at  Important  region  boundaries.  In  the  fol¬ 
lowing  figures,  the  borders  are  black  while  the  Interiors  are  constant  at  the  mean 
of  the  maximum  and  minimum  values  of  the  bands. 

Figure  6  shows  the  segmentation  of  the  original,  unsmoothed  Image  using  a 
centlle  threshold  of  75,  and  the  result  Is  as  poor  as  expected. 
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After  two  Iterations  of  color-SNN  smoothing  the  segmentation  Is  much 
better.  Figure  7a  shows  the  result  for  the  threshold  75,  and  Figure  7b  for  the 
threshold  80.  A  further  slight  Improvement  Is  obtained  by  five  Iterations,  with 
the  result  at  the  threshold  80  shown  In  Figure  8. 

Even  more  details  can  be  segmented  If  the  edges  are  enhanced  by  a  postpro¬ 
cessor  after  color-SNN  smoothing.  In  our  examples  we  used  the  MINRANGE 
filter  described  In  Section  4.  Figure  9a  shows  the  segmentation  after  two  Itera¬ 
tions  of  color-SNN  smoothing  and  two  Iterations  of  sharpening  with  MIN¬ 
RANGE,  and  Figure  9b  the  segmentation  after  five  Iterations  of  color-SNN 
smoothing  and  three  Iterations  of  sharpening.  The  threshold  value  Is  80. 

In  order  to  further  study  the  effects  of  Iterations  and  sharpening  we  did 
experiments  with  a  higher  threshold,  85,  which  should  cause  unwanted  merging. 
In  this  case,  two  or  five  Iterations  of  color-SNN  smoothing  without  sharpening 
did  not  give  very  good  results  (Figures  10a  and  10b).  The  results  for  two  Itera¬ 
tions  of  color-SNN  with  two  Iterations  of  sharpening  (Figure  10c)  and  for  five 
Iterations  of  color-SNN  with  three  Iterations  of  sharpening  (Figure  lOd)  are  very 
good.  These  and  other  experiments  we  performed  suggest  that  2-3  Iterations  of 
color-SNN  followed  by  2-3  Iterations  of  sharpening  Is  easily  sufficient  for  good 
results:  very  few  changes  In  segmentation  occur  with  more  Iterations. 

In  another  example  we  used  the  red  and  blue  bands  of  an  Image  of  a  room 
(Figure  11).  The  size  of  the  Images  Is  256  by  250  pixels  with  eight  bits  per  pixel. 
The  segmentation  of  the  original  bands  Is  shown  In  Figure  12a,  the  results  after 
two  Iterations  of  color-SNN  In  Figure  12b,  and  after  two  Iterations  of  color-SNN 
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followed  by  one  Iteration  of  sharpening  In  Fl  ure  12c.  The  threshold  value  Is  85. 
The  segmentation  of  the  original  Image  Is  poor,  while  the  other  segmentations  are 
very  good. 

Figure  13a  shows  the  color  edges  computed  for  the  original,  unsmoothed 
two-band  color  Image.  Figure  13b  shows  the  edges  after  two  Iterations  of  color- 
SNN.  and  Figure  13c  after  two  iterations  of  color-SNN  followed  by  two  of  MIN- 
RANGE  sharpening.  It  can  be  seen  that  the  color-SNN  alone  gives  good  results, 
and  that  the  effect  of  the  sharpening  Is  not  important  for  this  Image. 

The  methods  were  also  tested  on  a  set  of  road  Images  digitized  from  a  video¬ 
tape.  Figures  14a  and  14b  show  the  green  and  blue  bands  of  a  road  Image,  which 
has  a  highly  textured  background  and  a  shadow  on  the  road.  The  segmentation 
of  the  shadow  is  a  very  difficult  task.  The  size  of  the  Image  Is  128  by  128  pixels 
with  eight  bits  per  pixel.  Figures  14c  and  I4d  show  the  same  bands  after  three 
Iterations  of  color-SNN  smoothing.  The  textured  areas  have  become  significantly 
smoother. 

Figure  15a  shows  the  segmentation  (threshold  80)  after  three  Iterations  of 
color-SNN  followed  by  two  Iterations  of  sharpening  and  Figure  15b  shows  the 
segmentation  without  sharpening.  Figure  15c  shows  the  color  edges  for  the  case 
of  Figure  15a,  and  Figure  15d  for  the  case  of  Figure  15b,  with  a  threshold  of  95. 

In  our  previous  examples  we  used  Images  having  eight  bits  per  pixel.  The 
size  of  the  difference  histogram  Is  258  by  256  bins  for  8-blt  data.  We  also  did 
comparative  experiments  with  8-blt  data.  In  which  case  the  histogram  has  84  by 
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6-4  bins.  The  results  were  almost  Identical,  even  though  In  the  latter  case  nearly 


SO  percent  of  the  absolute  differences  were  In  the  nine  bins  nearest  to  the  origin 
for  a  smoothed  Image.  This  suggests  that  a  three-band  version  of  our  approach 
could  be  easily  Implemented  by  using  6-blt  data  or  by  using  adaptively  quantized 
histograms. 


7.  Conclusions 

A  new  measure  of  edge  Information  for  color  Images  based  on  cumulative 
histograms  of  absolute  color  differences  was  Introduced.  Methods  based  on  this 
measure  were  developed  for  edge-preserving  smoothing,  segmentation,  and  edge 
detection.  The  methods  performed  well  In  experiments.  Among  the  good 
features  of  the  methods  are  that  they  are  relatively  simple,  they  do  not  possess 
many  parameters,  and  they  seem  to  give  good  results  for  many  different  types  of 
Images  even  when  using  the  same  set  of  parameter  values  for  these  different 
Images. 
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FIGURE  CAPTIONS 


Figure  1.  Histogram  of  absolute  gray  level  differences. 

Figure  2.  Determination  of  cumulative  2-color  frequencies. 

Figure  3.  Original  bands  of  house  Image. 

a)  Red 

b)  Green 

Figure  4.  Bands  after  two  Iterations  of  color-SNN  smoothing. 

a)  Red 

b)  Green 

Figure  5.  Bands  after  five  iterations  of  color-SNN  smoothing. 

a)  Red 

b)  Green 

Figure  G.  Segmentation  of  original  bands  (threshold  75). 

Figure  7.  Segmentation  after  two  Iterations  of  color-SNN: 
a)  threshold  75  b)  threshold  80 

Figure  8.  Segmentation  after  five  Iterations  of  color-SNN  (threshold  80) 

Figure  9.  Segmentation  after  two  Iterations  of  color-SNN  and  two  Iterations  of 
sharpening  (a),  and  after  five  Iterations  of  color-SNN  and  three  Iterations  of  shar¬ 
pening  (b);  threshold  80. 

Figure  10.  Segmentation  with  threshold  85: 

a)  two  Iterations  of  color-SNN  b)  five  Iterations  of  color-SNN 

c)  two  Iterations  of  color-SNN  d)  five  Iterations  of  color-SNN 

and  two  Iterations  of  sharpening  and  three  Iterations  of  sharpening 
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Figure  11.  Red  and  blue  bands  of  room  Image. 

a)  Red 

b)  Blue 

Figure  12.  Segmentation  with  threshold  85: 

a)  for  original  bands 

b)  after  two  Iterations  of  color-SNN 

c)  after  two  Iterations  of  color-SNN 
and  one  Iteration  of  sharpening 

Figure  13.  Color  edges  of  house  Image,  with  threshold  90: 

a)  for  original  bands 

b)  after  two  Iterations  of  color-SNN 

c)  after  two  Iterations  of  color-SNN 
and  two  Iterations  of  sharpening 

Figure  14.  A  road  Image. 

a)  green 

b)  blue 

c)  green  after  three  Iterations  of  color-SNN 

d)  blue  after  three  iterations  of  color-SNN 

Figure  15. 

a)  Segmentation  with  threshold  80  after  three  Iterations 
of  color-SNN  and  two  Iterations  of  sharpening 

b)  Segmentation  without  sharpening 

c)  Edges  for  (a)  with  threshold  95 

d)  Edges  for  (b)  with  threshold  95 
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Figure  3.  --  Original  bands  of  house  Image. 
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Figure  7.  --  Segmentation  after  2  Iterations  of  color-SNN. 
a)  threshold  75  b)  threshold  80 
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Figure  9.  —  Segmentation  arter  2  Iterations  of  color-SNN  and  2  Iterations  of 
sharpening  (a),  and  after  5  Iterations  of  color-SNN  and  3  Iterations  of  sharpening 
(b),  (threshold  80). 


Figure  10.  —  Segmentation  with  threshold  85. 

a)  2  Iterations  of  eolor-SNN  b)  5  Iterations  of  color-SNN 

c)  2  Iterations  of  color-SNN  d)  5  Iterations  of  color-SNN 


+  2  Iterations  of  sharpening  4-3  Iterations  of  sharpening 


L 


a) 


Figure  12.  -  Segmentation  with  threshold  85. 

a)  for  original  bands 

b)  after  2  Iterations  of  color-SNN 

c)  after  2  Iterations  of  coior-SNN 
and  1  Iteration  of  sharpening 
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Figure  13.  —  Color  edges  of  house  Image, 
with  threshold  90. 

a)  for  original  bands 

b)  after  2  Iterations  of  color-SNN 

c)  after  2  Iterations  of  color-SNN 
and  2  Iterations  of  sharpening 
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Figure  14.  —  A  road  Image. 

a)  green 

b)  blue 

c)  green  after  3  Iterations  of  color-SNN 

d)  blue  after  3  Iterations  of  color-SNN 


Figure  15. 

a)  Segmentation  with  threshold  80  after  3  Iterations  of  color-SNN 
+  2  Iterations  of  sharpening 

b)  Segmentation  without  sharpening 

c)  Edges  for  (a)  with  threshold  95 

d)  Edges  for  (b)  with  threshold  95 
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